The present invention relates to a calorimetric apparatus and method of classifying, determining, or identifying a target object by using colors.
Conventionally, a color classification apparatus for classifying colors of target objects is used in management of painting colors and the degree of dyeing, colorimetry of products at production sites in various industries, or colorimetry of specimens in medical and scientific fields.
As a color classification apparatus of this type, an inexpensive apparatus having a simple arrangement and resistance to mechanical vibrations is desired.
Particularly, although color classification in, e.g., a factory can be performed while limiting the light source, an apparatus capable of satisfactorily classifying colors even when the light source is not limited, i.e., even when the spectrum changes, is desired.
U.S. Ser. No. 08/321,667, which has issued as U.S. Pat. No. 5,717,605 and which is referred to herein as the "prior application", discloses an invention which relates to an inexpensive color classification apparatus having a simple arrangement and resistance to mechanical vibrations and capable of satisfactorily classifying colors even when the light source is not limited, i.e., even when the spectrum changes.
The color classification apparatus disclosed in this prior application is characterized by comprising an image pickup means for picking up the reflection spectrum of a target object, a plurality of bandpass filters arranged between the target object and the image pickup means and having different bands, and a classification means for calculating a classification spectrum for classification using a statistical method from the reflection spectrum of the target object whose image is picked up by the image pickup means and classifying the target object by using the classification spectrum.
According to the color classification apparatus of this prior application, the plurality of bandpass filters having different bands are prepared and arranged between the target object and the image pickup means.
The classification means calculates the classification spectrum for classification using the statistical method from the reflection spectrum of the target object whose image is picked up by the image pickup means and classifies the target object by using this classification spectrum.
The principle of the color classification apparatus of the prior application will be described first.
The color classification apparatus of the prior application uses a filter as shown in FIGS. 22B or 22C which combines a plurality of bandpass filters for transmitting specific wavelengths as shown in FIG. 22A, thereby realizing an inexpensive color classification apparatus having a simple arrangement.
To perform color classification even under different light sources, the color classification apparatus of the prior application measures the reflection spectrum of an appropriate reference plate under the same conditions as those in picking up the image of the target object and corrects the reflection spectrum of the target object with the reflection spectrum of the reference plate, thereby removing the influence of the light source (illumination light).
More specifically, when the wavelength is represented by .lambda.; the reflection spectrum of the target object, f(.lambda.); the reflection spectrum of the reference plate, s(.lambda.); the reflection spectrum of the illumination light, L(.lambda.); and the sensitivity spectrum of the photographing system (e.g., the transmittance spectrum of the photographing lens and the sensitivity spectrum of the photographing device), M(.lambda.), a photographic spectrum gi(.lambda.) of the target object and a photographic spectrum gs(.lambda.) of the reference plate are represented by equations below, respectively: EQU gi(.lambda.)=f(.lambda.).times.L(.lambda.).times.M(.lambda.) EQU gs(.lambda.)=s(.lambda.).times.L(.lambda.).times.M(.lambda.)
A spectrum gi'(.lambda.) of the target object can be represented by equation (1): EQU gi'(.lambda.)=gi(.lambda.)/gs(.lambda.)=f(.lambda.)/s(.lambda.)(1)
With this method, the color classification apparatus of the prior application can remove the influence of the reflection spectrum L(.lambda.) of the illumination light. Using gi'(.lambda.), target objects can be classified under different light sources.
When the luminance of illumination light changes, the power of the signal gi'(.lambda.) after removal of the influence of the reflection spectrum L(.lambda.) of the illumination light may be normalized.
The color classification apparatus of the prior application which classifies target objects of two classes will be described below.
FIG. 21 is a view showing the arrangement of the color classification apparatus. The color classification apparatus of the prior application comprises an optical system 10 containing a stop and a lens, a rotary color filter 12 constituted by a plurality of bandpass filters 12A to 12E as shown in FIG. 22A, a CCD 14 for picking up the images of a target object O and a reference plate R, an A/D converter 16, a frame memory 18, a monitor 20 for displaying a portion which is being photographed, a CCD driver 22, a driving motor 24 for the rotary color filter 12, a control unit 26 for controlling the CCD driver 22, the driving motor 24, and the like and also sending a command to a classification arithmetic circuit 28, and the classification arithmetic circuit 28 for performing classification.
As shown in FIG. 22B, the rotary color filter 12 is constituted by the bandpass filters 12A to 12E of several types. Each filter has characteristics for transmitting an arbitrary bandwidth as shown in FIG. 22A.
For descriptive and illustrative convenience, an example wherein the rotary color filter 12 is constituted by five bandpass filters will be explained.
As for the arrangement of the optical system 10 and the rotary color filter 12, the rotary color filter 12 may be placed before the optical system 10.
As shown in FIG. 23, the classification arithmetic circuit 28 comprises a luminance component extraction unit 30 for extracting the luminance component of the target object O, a classification arithmetic unit 32 for performing calculation (e.g., FS transform) for classification, and a classification determination unit 34 for performing learning for classification determination and performing classification determination.
As shown in FIG. 24, the luminance component extraction unit 30 has three measurement area extraction units 36A, 36B, and 36W for extracting the measurement areas of the target object O and the reference plate R from the photographed images, three luminance component averaging units 38A, 38B, and 38W for calculating the averages of the measured luminance components, a luminance component memory "A" 40A in which the luminance component of photographed data of class 1 or unknown class is written, a luminance component memory "B" 40B in which the luminance component of photographed data of class 2 is written, a luminance component memory "W" 40W in which the luminance component of photographed data of the reference plate R is written, a correction circuit 42 for correcting the influence of the light source, a luminance spectrum memory "dta" 44A in which the corrected data of class 1 or unknown class is written, and a luminance spectrum memory "dtb" 44B in which the corrected data of class 2 is written.
Luminance components corresponding to the number of bandpass filters (five in this example) constituting the rotary color filter 12 can be written in the luminance component memories 40A, 40B, and 40W.
The correction circuit 42 is constituted by a divider 42A, as shown in FIG. 25A, or the divider 42A and a power normalization circuit 42B, as shown in FIG. 25B.
The arrangement shown in FIG. 25B will be described below.
Luminance components (each luminance component consists of data corresponding to the number of filters) corresponding to a sample count N of data to be photographed can be written in the luminance spectrum memories 44A and 44B.
As shown in FIG. 25C, the classification arithmetic unit 32 comprises a change-over switch "A" 46, a classification spectrum calculation unit 48 for calculating the classification spectrum, a classification spectrum d1 memory 50 in which a classification spectrum d1 is written, a classification spectrum d2 memory 52 in which a classification spectrum d2 is written, a change-over switch "B" 54, a multiplier 56, and an accumulation unit 58 constituted by an adder 58A and a latch circuit 58B to perform accumulation.
As shown in FIG. 25C, the classification determination unit 34 comprises a change-over switch "C" 60, a classification boundary determination unit 62 for determining the classification boundary, a classification boundary memory "c1" 64 in which the determined classification boundary is written, and a classification determination unit 66 for performing classification determination.
Processing of classifying target objects of two classes by using the color classification apparatus having the above arrangement will be described next.
In this processing, a learning mode is executed first to obtain the classification boundary, and a classification mode is executed next to perform color classification for data of unknown class.
The learning mode will be described first.
In the learning mode, a classification spectrum for classifying the target objects O of two classes as shown in FIG. 26 is obtained.
The control unit 26 adjusts the direction and focal length of the optical system 10 to simultaneously pick up the images of the target objects of two classes.
Focus adjustment is performed by a focus adjustment mechanism (not shown). At the same time, photometry is performed by a photometer (not shown), and the stop of the optical system 10 and the exposure time of the CCD 14 are set.
The position of the rotary color filter 12 is controlled such that the photographing operation is performed through the first bandpass filter (e.g., 12A) of the rotary color filter 12.
When a photographing command is sent to the CCD driver 22, the first image is picked up.
The image data picked up by the CCD 14 is A/D-converted by the A/D converter 16, transferred to the frame memory 18, and stored in the frame memory 18.
The image data stored in the frame memory 18 is loaded in the classification arithmetic circuit 28.
The classification arithmetic circuit 28 transfers the image data to the luminance component extraction unit 30.
In the luminance component extraction unit 30, the measurement area extraction units 36A and 36B extract classification target areas for classes 1 and 2, respectively, from the loaded image data and extract luminance components in units of pixels of the classification target areas.
The luminance component averaging units 38A and 38B detect the average values of luminances in the respective areas. The detected average values are written in the luminance component memories 40A and 40B.
These values will be referred to as data da1 and db1.
Next, the rotary color filter 12 is rotated to pick up the second image through the second filter (e.g., 12B), and in a similar fashion, average values are written in the luminance component memories 40A and 40B.
These values will be referred to as data da2 and db2.
The same operation is repeated to the fifth filter (e.g., 12E) so that data da3, da4, and da5 are written in the luminance component memory "A" 40A, and data db3, db4, and db5 are written in the luminance component memory "B" 40B.
That is, in the series of operations, data dai (i=1 to 5) are written in the luminance component memory "A" 40A, and data dbi (i=1 to 5) are written in the luminance component memory "B" 40B.
Next, the reference plate R is set near the target objects, and the image of the reference plate R is photographed through the five filters, as described above, and data dwi (i=1 to 5) are written in the luminance component memory "W" 40W.
The correction circuit 42 reads out data, for class 1, from the luminance component memory "A" 40A and the luminance component memory "W" 40W, and for class 2, from the luminance component memory "B" 40B and the luminance component memory "W" 40W and corrects the data.
This correction is performed by dividing the data from the luminance component memory "A" 40A by the data from the luminance component memory "W" 40W in units of filter components by the divider 42A in accordance with equations (2a) below: EQU da.sup.m i'=da.sup.m i/dwi(for i=1 to 5, m=1 to N) EQU db.sup.m i'=db.sup.m i/dwi(for i=1 to 5, m=1 to N) (2a)
Where i is the filter number, and m is the sample number.
With this calculation, the influence of different light sources (spectral characteristics) can be removed.
The power normalization circuit 42B calculates equations (2b) below using power values Ca.sup.m and Cb.sup.m such that the power values of the data divided by the correction circuit 42 are equalized: EQU da.sup.m i"=da.sup.m i'/Cam(for i=1 to 5, m=1 to N) EQU db.sup.m i"=db.sup.m i'/Cbm(for i=1 to 5, m=1 to N)
where the power values Ca.sup.m and Cb.sup.m are given by equations (2c): ##EQU1## or equations (2d): ##EQU2##
With this power normalization, the influence of a change in luminance of the light source can be removed.
Resultant da.sup.m i" and db.sup.m i" are written in the luminance spectrum memory "dta" 44A and the luminance spectrum memory "dtb" 44B, respectively, as luminance spectra.
The above correction is repeated a number of times corresponding to the sample count N of the target objects, and luminance spectra are written in the luminance spectrum memory "dta" 44A and the luminance spectrum memory "dtb" 44B.
For the sample, the target object itself may be exchanged, or different areas of the same target object may be used.
With this processing, luminance spectrum data corresponding to the sample count N of the target objects are written in the luminance spectrum memory "dta" 44A and the luminance spectrum memory "dtb" 44B.
If the target objects of two classes cannot be simultaneously photographed, the photographing and correction operations are performed for the target object and reference plate in units of classes, and the luminance spectra are written in the luminance spectrum memory "dta" 44A and the luminance spectrum memory "dtb" 44B.
This operation is performed a number of times corresponding to the sample count N.
In the classification arithmetic unit 32, the change-over switch "A" 46 is switched to the terminal b side.
The luminance spectrum data associated with classes 1 and 2 are read out from the luminance spectrum memory "dta" 44A and the luminance spectrum memory "dtb" 44B, respectively. The classification spectrum calculation unit 48 obtains a classification spectrum d1i (i=1 to 5) and a classification spectrum d2i (i=1 to 5) perpendicular to the classification spectrum d1i by using FS transform.
The classification spectrum d1i and classification spectrum d2i obtained in the above manner are written in the classification spectrum d1 memory 50 and the classification spectrum d2 memory 52, respectively.
The change-over switch "A" 46 is switched to a terminal a side, and simultaneously, the change-over switch "C" 60 of the classification determination unit 34 is switched to the terminal b side.
The change-over switch "B" 54 is switched to the terminal a side. The luminance spectrum data da.sup.m i" and the classification spectrum data dli are read out from the luminance spectrum memory "dta" 44A and the classification spectrum d1 memory 50, respectively. The inner product is calculated by the multiplier 56 and the accumulation unit 58: ##EQU3## The calculation result is transferred to the classification boundary determination unit 62 of the classification determination unit 34.
The luminance spectrum data db.sup.m i" and the classification spectrum data dl1 are read out from the luminance spectrum memory "dtb" 44B and the classification spectrum d1 memory 50, respectively. The inner product is calculated as described above: ##EQU4## The calculation result is transferred to the classification boundary determination unit 62.
The change-over switch "B" 54 is switched to the terminal b side. The luminance spectrum data da.sup.m i" and the classification spectrum d2i are read out from the luminance spectrum memory "dta" 44A and the classification spectrum d2 memory 52, respectively. The inner product is calculated: ##EQU5## The calculation result is transferred to the classification boundary determination unit 62.
The luminance spectrum data db.sup.m i" and the classification spectrum d2i are read out from the luminance spectrum memory "dtb" 44B and the classification spectrum d2 memory 52, respectively. The inner product is calculated: ##EQU6## The calculation result is transferred to the classification boundary determination unit 62.
The above processing is performed a number of times corresponding to the sample count for each class.
Using the obtained inner product values, the classification boundary is determined by the classification boundary determination unit 62, as shown in FIG. 25, and written in the classification boundary memory "c1" 64.
The learning mode has been described above.
The classification mode will be described next.
In the classification mode, the target objects O of an unknown class to be classified as shown in FIG. 26 are photographed, as in the learning mode, and a luminance spectrum dxi (i=1 to 5) is written in the luminance component memory "A" 40A.
Subsequently, the reference plate. R is photographed under the same photographing conditions, and the luminance spectrum dwi (i=1 to 5) is written in the luminance component memory "W" 40W.
The data are read out from the luminance component memories "A" 40A and "W" 40W and corrected by the correction circuit 42 on the basis of the equation below: EQU dxi"=dxi'/Cx(for i=1 to 5)
In addition, the following equations are calculated such that the power value of the divided data is normalized by the power normalization circuit 42B: ##EQU7##
A spectrum dxi" is written in the luminance spectrum memory "dta" 44A.
The change-over switch "A" 46 of the classification arithmetic unit 32 is switched to the terminal a side, and the change-over switch "C" 60 of the classification determination unit 34 is switched to the terminal a side.
The change-over switch "B" 54 of the classification arithmetic unit 32 is switched to the terminal a side. The luminance spectrum dxi" and the classification spectrum data d1i are read out from the luminance spectrum memory "dta" 44A and the classification spectrum d1 memory 50, respectively. The inner product is calculated by the multiplier 56 and the accumulation unit 58: ##EQU8## A value tx1 is transferred to the classification determination unit 66 of the classification determination unit 34.
The change-over switch "B" 54 is switched to the terminal b side. The luminance spectrum data dxi" and the classification spectrum data d2i are read out from the luminance spectrum memory "dta" 44A and the classification spectrum d2 memory 52, respectively. The inner product is calculated: ##EQU9## A value tx2 is transferred to the classification determination unit 66.
The classification determination unit 66 reads out the classification boundary from the classification boundary memory "c1" 64, determines, on the basis of the readout data, on which side of the classification boundary the transferred inner product values tx1 and tx2 exist, and outputs the classification result.
The operation described above is the classification mode.
As described above, in the color classification apparatus of the prior application, the difference in the spectral characteristic between light sources is corrected by the divider 42A, and the difference between luminances is corrected by the power normalization circuit 42B. For this reason, satisfactory classification can be performed even under different light sources.
In addition, since the power normalization circuit 42B is used, as shown in FIG. 25B, satisfactory classification can be performed even in case of a change in luminance of the light source.
If not the spectrum but only the luminance of the light source changes, the divider 42A can be omitted, and the power normalization circuit 42B suffices.
This apparatus has a simple arrangement using the rotary color filter 12 and is therefore inexpensive and resistible to mechanical vibrations.
The apparatus readily copes with different classification purposes because it has the learning and classification modes.
Alternatively, as shown in FIG. 29, the classification arithmetic unit 32 may have a plurality of sets of pairs of the classification spectrum d1 and d2 memories 50 and 52 for storing the already learned classification spectra d1i and d2i and change-over switches "B" 54 for selecting one of the pairs of classification spectrum d1 and d2 memories 50 and 52. With this arrangement, when different learned classification spectra are stored in the classification spectrum d1 and d2 memories 50 and 52 of each pair, and a change-over switch "C'" 68 is used to select the pair of the classification spectrum d1 and d2 memories 50 and 52, the apparatus can instantaneously cope with different classification purposes.
In this example, the rotary color filter 12 has a structure in which the circular filters 12A to 12E are arranged on one circle, as shown in FIG. 22B. To stop the rotary color filter 12 at each filter, the position of the rotary color filter 12 is controlled in units of filters. When the filters 12A to 12E are formed into circular arcs and arranged on one circle to constitute the rotary color filter 12, position control for stopping the rotary color filter 12 at each filter need not be performed, so that the rotary color filter 12 can be continuously moved. With this arrangement, color classification processing for target objects can be performed at a higher speed.
In this case, however, the exposure timing of the CCD 14 must be synchronized with the rotation speed of the rotary color filter 12, as a matter of course.
The classification result may be displayed as images of different colors in accordance with classes or informed to the photographer with sounds.
In the color classification apparatus of the prior application the color components of the target objects are classified on the basis of multispectrum image data which is obtained by picking up the images of the target objects through the rotary filter having the plurality of optical bandpass filters with different pass band characteristics.
In the color classification apparatus of the prior application the plurality of optical bandpass filters to be used are optimized in accordance with the target object. When the target object changes, or an unknown target is used, the plurality of optical bandpass filters must be exchanged, so room is left for an improvement in versatility.
In addition, for the color classification apparatus of this prior application, a specific technique of optimizing the plurality of measurement filters which must be prepared in advance as filters for measuring the color of the target object is neither disclosed nor proposed.
In the prior application an apparatus having a means for exchanging the plurality of optical bandpass filters in accordance with the light source is disclosed, although no filter exchanging means for various target objects is described.
The present inventors have also filed in another application, U.S. Ser. No. 08/701,123, which is referred to as the "second prior application", a highly versatile color classification apparatus which is applicable to various target objects and comprises an image pickup means for picking up the image of light from a target object, an optical means for allowing the light from the target object to form an image on the image pickup means, a plurality of optical bandpass filters arranged between the target object and the image pickup means and having different pass bands, a switching means for switching the plurality of optical bandpass filters, and an arithmetic processing means for performing classification or determination using a statistical method on the basis of the spectral characteristics of the photographed target object, wherein the plurality of optical bandpass filters include a plurality of measurement filters for measuring the colors of the target object, and an inspection filter for determining, from the plurality of measurement filters, a filter to be applied to the target object, and the switching means switches/selects the effective measurement filter on the basis of the result detected using the inspection filter.
However, the second prior application, like the above described first prior application, does not disclose or propose a specific technique of optimizing the plurality of measurement filters which must be prepared in advance as filters for measuring the color of the target object.
That is, when the plurality of measurement filters themselves which must be prepared in advance are optimized, effective color classification can be immediately performed without switching/selecting the effective measurement filter on the basis of the result detected using the inspection filter. This, however, is not achieved by the color classification apparatus of the second prior application.